Decoding visual codes

ABSTRACT

Various algorithms are presented that enable an image of a data matrix to be analyzed and decoded for use in obtaining information about an object or item associated with the data matrix. The algorithms can account for variations in position and/or alignment of the data matrix. In one approach, the image is analyzed to determine a connected region of pixels. The connected region of pixels can be analyzed to determine a pair of pixels, included in the connected region of pixels, that is separated a greatest distance wherein a first pixel and second pixel of the pair of pixels is associated with image coordinates. Using the image coordinates of the pair of pixels, a potential area of the image that includes the visual code can be determined and the potential area can be analyzed to verify the presence of a potential data matrix.

This application is a continuation of U.S. application Ser. No.15/282,829 entitled “DECODING VISUAL CODES,” filed Sep. 30, 2016, whichis a continuation of U.S. application Ser. No. 14/799,327 entitled“DECODING VISUAL CODES,” filed Jul. 14, 2015, now U.S. Pat. No.9,501,681, issued Nov. 22, 2016, which is incorporated herein byreference for all purposes.

BACKGROUND

As portable electronic devices offer an ever-increasing amount offunctionality, people are using these devices to assist with a greatervariety of tasks. As an example, a person in a store might want toobtain information about an item, such as reviews for a book or a pricefrom a retailer of that item. Some applications enable a user to capturean image of a quick response (QR) code on an item, and upload that imageto a central server where the image can be analyzed and decoded. Such anapproach can require a significant amount of time and bandwidth toupload, and requires an acceptable cellular signal in many instances,which can be difficult to obtain in large brick and mortar stores orother such locations. While the functionality could be loaded onto thedevice, conventional systems typically require the QR code to be wellaligned and positioned in the center of the image, which can make itdifficult to decode these images using conventional algorithms, thusrequiring a significant amount of processing capacity and powerconsumption, which might be unavailable or at least undesirable for auser of a small portable device.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIGS. 1A and 1B illustrate an example situation wherein a user capturesan image of a visual code using a portable electronic device, which isthen displayed on the electronic device;

FIGS. 2A-2K illustrate an example of analyzing visual codes inaccordance with various embodiments;

FIG. 3 illustrates an example process for decoding a visual code from animage in accordance with various embodiments;

FIG. 4 illustrates front and back views of an example device that can beused in accordance with various embodiments;

FIG. 5 illustrates an example configuration of components of a devicesuch as that described with respect to FIG. 4; and

FIG. 6 illustrates an example environment in which various aspects ofthe various embodiments can be implemented.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of thepresent disclosure may overcome one or more of the aforementioned andother deficiencies experienced in conventional approaches to obtaininginformation from a visual code such as a data matrix. In particular,various embodiments provide for recognizing and decoding data matricesfrom a captured image that are positioned at an arbitrary positionand/or orientation in the image. An image including a representation ofa data matrix can have been captured using an image capture element ofany appropriate device, such as a digital camera of a portable computingdevice. In some situations, the data matrix may be positioned such thatit is difficult to align and orient the data matrix in the center orother appropriate area of the image. Further, a device such as a tabletcomputer or smart phone often is held in a user's hand, and thus mightbe moving slightly while capturing the image, which can make itdifficult to align and orient the data matrix. Further still, any of anumber of other reasons might lead to an arbitrary position and/ororientation of the data matrix in the captured image, which may make itnot possible or at least difficult for conventional data matrix readingalgorithms to provide an accurate result in at least some cases.

Accordingly, a decoding process in accordance with at least oneembodiment includes analyzing an image to identify a data matrixregardless of the data matrix's position and/or orientation in theimage. In one such approach, the image is analyzed to determine aconnected region of pixels. The connected region of pixels can beanalyzed to determine a pair of pixels, included in the connected regionof pixels, that is separated a greatest distance wherein a first pixeland second pixel of the pair of pixels is associated with imagecoordinates. Using the image coordinates of the pair of pixels, apotential area of the image that includes the visual code can bedetermined and the potential area can be analyzed to verify the presenceof a potential data matrix. In the situation where the presence of thedata matrix is verified, the data matrix in the area is decoded. Variousother approaches and implementations can be used as well as discussedelsewhere herein.

In accordance with various embodiments, algorithms can also beconfigured to handle variations of specific data matrix standards, aswell as multiple data matrix formats. At least some of these algorithmsalso can be executed quickly enough, and with a minimal amount ofresources, such that in at least some embodiments the entire data matrixreading and decoding process can be performed on the portable computingdevice. In other embodiments, a captured image or other such informationcan be sent to a remote server or other device for analysis, which thencan return information relating to the barcode to the device.

Various other applications, processes and uses are presented below withrespect to the various embodiments.

FIG. 1A illustrates an example environment 100 wherein a user 102 isoperating an electronic device 104 that includes at least one imagecapture element. In this example, the user 102 locates an item or objectof interest 106. If the user wants to obtain information about the item,such as a description or pricing information, the user 102 canmanipulate the electronic device 104 until a viewable area 110 of acamera of the device includes a data matrix 108 positioned on, attachedto, or otherwise associated with, the item of interest 108. Inaccordance with various embodiments, a data matrix is a two-dimensionalmatrix barcode consisting of black and white “cells” or modules arrangedin either a square or rectangular pattern. The information to be encodedcan be text or numeric data. In this example, the data matrix iscomposed of two solid adjacent borders in an “L” shape (called the“finder pattern”) and two other borders consisting of alternating darkand light “cells” or modules (called the “timing pattern”). Within theseborders are rows and columns of cells encoding information.

FIG. 1B illustrates an example of the data matrix 154 when displayed aspart of the captured image on a display element 152 of an electronicdevice 150. As described, some applications enable a user to capture animage of the data matrix to analyze and decode the data matrix. However,such conventional approaches typically require the data matrix to bealigned and positioned in the center of the image before the data matrixcan be decoded. In some situations, such as the one shown in FIG. 1B, abounding window to which the data matrix is to be aligned can beprovided to help assist the user in capturing a useful image. The usercan attempt to manipulate the electronic device until a viewable area ofa camera of the device includes the data matrix positioned on, attachedto, or otherwise associated with, the item of interest is properlycentered and aligned. However, in some situations, this process can bedifficult and time consuming, which can be frustrating to a user wherethe user. As shown in FIG. 1B, the data matrix is not properly orientedor aligned as the top of the data matrix is titled clockwise and thedata matrix is partially positioned outside the bounding window. In thissituation, the device may not be able to decode the data matrix.Accordingly, various embodiments provide for recognizing and decodingdata matrices from a captured image that are positioned at an arbitraryposition and/or orientation in the image.

FIGS. 2A-2H illustrate an example approach for identifying visual codesfrom a captured image that are positioned at an arbitrary positionand/or orientation in the image. FIG. 2A illustrates a view 200 of theexample visual code displayed on the device in FIG. 1B. In this example,the visual code is a data matrix (e.g., a QR code). The image includesother items other than the data matrix. As described, if a user wants toobtain information about an item, such as a description or pricinginformation, the user can manipulate the electronic device until aviewable area of a camera of the device includes the data matrixpositioned on, attached to, or otherwise associated with, the item ofinterest. In some situations, the image can be downloaded or obtained ina way other than capturing an image of the item of interest.

The data matrix is a two-dimensional matrix barcode consisting of blackand white “cells” or modules arranged in either a square or rectangularpattern. Depending on the coding used, a “light” cell represents a 0 anda “dark” cell is a 1, or vice versa. As used herein, the black and whitesquares will be referred to as modules rather than pixels todifferentiate between on-screen pixels and the black and white squaresof the data matrix. For example, a data matrix may be 21 modules by 21modules, even if it takes up 42 by 42 pixels on a computer screen, or105×105, and so on. A data matrix can include a number of functionpatterns. A function pattern can be, for example, shapes placed inspecific areas of the data matrix to ensure that data matrix scannerscan correctly identify and orient the data matrix for decoding.

Example function patterns are illustrated in FIG. 2B. Example 210 ofFIG. 2B illustrates a finder pattern (214, 216) and a timing pattern(212, 218). The finder pattern can be used to locate and orient the datamatrix while the timing pattern provides a count of the number of rowsand columns in the data matrix. In this example, the finder patternconsists of two solid adjacent borders in an “L” shape, while the timingpattern consists of adjacent borders with alternating dark and lightcells. As should be understood, other data matrix formats can utilizedifferent patterns as known in the art. As should be further understood,other function patterns, in addition to or instead of those described,can be included in the data matrix. Example patterns include finderpatterns, separators, alignment patterns, dark modules, among others.

Once the image is obtained, a detection component or other suchcomponent(s) can attempt to determine whether the data matrix associatedwith the item of interest is identifiable. In accordance with variousembodiments, the image can be binarized to generate a binarized image.Those skilled in the art will understand that in certain embodiments,the image is not binarized, and that the approaches described herein canbe performed on various types of images of various formats. A binaryimage is a digital image that has only two possible values for eachpixel. Typically the two colors used for a binary image are black andwhite though any two colors can be used. As would be understood to oneskilled in the art, any one of a number of image binarizing approachescan be used to generate the binarized image. One approach is to usethresholding to convert a grayscale image into a binary black and whiteimage (such as by setting any pixels in the image above a giventhreshold to black and all others to white).

The binarized image (and in certain examples the obtained image (i.e.,the captured or downloaded image)) can be analyzed using at least oneconnected-component analysis approach to determine a plurality ofconnected-components. In accordance with various embodiments,connected-component analysis, connected-component labeling, blobextraction, region labeling, blob discovery, region extraction), amongother like approaches, are algorithmic applications of graph theory,where subsets of connected-components are uniquely labeled based on agiven heuristic. In one such approach, the computing device scans animage and groups its pixels into components based on pixel connectivity,i.e. all pixels in a connected-component share similar pixel intensityvalues and are in some way connected with each other. Once all groupshave been determined, each pixel is labeled with a gray level or a color(color labeling) according to the component it was assigned to. In thisway, connected-component analysis works by scanning an image,pixel-by-pixel (from top to bottom and left to right) in order toidentify connected pixel regions, i.e. regions of adjacent pixels whichshare the same set of intensity values.

An example connected-component is a subgraph in which any two verticesare connected to each other by paths, and which is connected to noadditional vertices in the supergraph. Those skilled in the art willunderstand that given a graph G=(V,E), where V is a set of vertices (ofsize n) and E is a set of edges (of size m), the connected-components ofG are the sets of vertices such that all vertices in each set aremutually connected (reachable by some path), and no two vertices indifferent sets are connected.

The connected-components are compared to a threshold connected-componentsize. Connected-components less than the threshold size are discarded orotherwise processed in a different manner. In accordance with anembodiment, the threshold connected-component size can correspond to,for example, a number of connected pixels. In this way, aconnected-component having less than a threshold number of connectedpixels is discarded or processed in a different manner. In anotherexample, the threshold size can correspond to the length of theconnected-component. As will be apparent to a person of reasonable skillin the art, the length of the connected-component can be determined in anumber of different ways. In one such way, a starting pixel in the imageis selected. From the starting pixel, each node is scanned until allnodes are exhausted. This is repeated until all connected-components aredetermined.

Once the connected-components are compared to the threshold size, one ormore connected-components will remain. Example 220 of FIG. 2Cillustrates one such connected-component 222. It should be noted thatfor purposes of explanation, FIG. 2C illustrates oneconnected-component. As would be understood to one skilled in the art,in various embodiments, a plurality of components that meet thethreshold size can be identified. It should also be understood thatalthough only connected-component 222 is shown, this is merely forpurposes of explanation, and the not shown portions of the data matrixare still used in the processing.

In this example, connected-component 222 is analyzed to determine twopixels, pixel A and pixel B, that are the furthest distance apart onconnected-component 222. As shown in example 230 of FIG. 2D, the twopixels correspond to points 232 and 236, which for the data matrixillustrated and assuming the image undergoes distortion less than athreshold amount, typically corresponds to the two ends of the “L”shape.

In accordance with an embodiment, determining two pixels that are thefurthest distance apart on the connected-component can include, forexample, determining a plurality of distance measurements for each pixelin connected-component 222, where each distance measurement of theplurality of distance measurements corresponds to a distance between apair of pixels in connected-component 222. For example, starting with aninitial pixel in the connected-component, the distance from the initialpixel to each of the other pixels can be determined. This process can berepeated from each pixel in the connected-component and will result in aplurality of distances. Thereafter, a greatest distance can be selectedfrom the plurality of distances. The greatest distance is associatedwith a first pixel (pixel A) and a second pixel (pixel B) inconnected-component 222, the first pixel having a first location on theimage and the second pixel having a second location on the image. Thefirst location and the second location, or any pixel location, should beunderstood to be defined by the x, y coordinates of each pixel on theimage. As would be understood to one skilled in the art, any one of anumber of digital image processing algorithms can be used to measure theEuclidean distance between pixels in an image.

Once the two pixels, pixel A and pixel B, that are the furthest distanceapart are determined, a third pixel, pixel C, is determined based on thelocation of pixel A and pixel B on the image. The location of pixel Ccan be determined by, for example, determining a pixel with a locationthat is the largest sum of distances to the location of pixel A andpixel B. As shown in example 240 of FIG. 2E, point C can correspond topoint 242, which for the data matrix illustrated and assuming the imageundergoes distortion less than a threshold amount, typically correspondsto the pixel located at the corner of the “L.”

In accordance with an embodiment, determining the location of pixel Ccan include, for example, determining, for each pixel inconnected-component 222, a sum of distance from a pixel in theconnected-component to the pixel A and to pixel B. The process isrepeated for each pixel in the connected-component, and the pixelcorresponding to the largest sum of distance is pixel C. In accordancewith various embodiments, pixel A, B, and C will each have a location onthe image. As would be understood to one skilled in the art, any one ofa number of ways can be used to express a location on an image In onesuch approach, the location of a pixel can be expressed using pixelindices. For example, the image can be treated as a grid of discreteelements, ordered from top to bottom and left to right. For pixelindices, the row increases downward, while the column increases to theright. Pixel indices are integer values, and range from 1 to the lengthof the row or column. As an example, the data for the pixel in the fifthrow, second column is stored in the matrix element (5,2).

Based at least in part on the locations of pixels A, B, and C, alocation of a fourth pixel, pixel D, can be determined. As illustratedin example 250 of FIG. 2F, pixel D can correspond to point 252. Thelocation of pixel D can be determined, for example, using the geometricrelationship between pixels A, B, and C. For example, assuming thelocations of A, B, C for a portion of a parallelogram, the location ofpixel D can be determined based on the geometric properties of aparallelogram. Various other image analysis approaches to determine thegeometric relationship between pixel locations on an image can be usedas well as should be apparent to one of ordinary skill in the art inlight of the present disclosure.

An area or region of interest can be determined using the locations ofpixels A, B, C, and D, where the region of interest potentially containsthe data matrix. In this way, in an image that identified one or moreregions of interest, each region would include a candidate data matrix.In accordance with various embodiments, a region of interest (ROI), forexample, can be an area of an image to focus image analysis.

A binary mask can be used to define the region of interest. As shown inexample 260 of FIG. 2G, a mask 262 is formed using the locations ofpixels A, B, C, and D. Once the binary mark is generated, an arbitraryimage matrix can be specified. In this example, the pixel values of theimage corresponding to binary mask can be the original pixel values, andall other pixel values can be some other pixel value. As shown inexample 270 of FIG. 2H, the region of interest 272 includes the pixelvalues of the data matrix. The pixel values outside 276 the data matrixcan be some other value (e.g., zero). In various embodiments, to ensurethe region of interest includes the data matrix, the region of interestcan be dilated by certain amount, as shown by region 274. The amount canbe, for example, 1-3 pixels.

The image can be analyzed using at least one image processing algorithmto locate the outermost pixels. As shown in example 280 of FIG. 2I, theoutermost pixels are in regions 282, 284, 286, and 288. For explanationpurposes, only pixels in those regions are displayed and pixels outsidethose regions are not shown. Locating the outermost pixels can include,for example, running scan lines in a particular direction (e.g., outsidethe data matrix towards the inside of the data matrix in a particulardirection) to locate the outermost pixels within the region of interest.In accordance with an embodiment, this can include, for example,scanning the first image in a first direction to determine a first setof pixels, the first set of pixels being positioned along a first pathbetween the first image coordinates and the second image coordinates;scanning the image in a second direction to determine a second set ofpixels, the second set of pixels being positioned along a second pathbetween the first image coordinates and the fourth image coordinates;scanning the image in a third direction to determine a third set ofpixels, the third set of pixels being positioned along a third pathbetween the fourth image coordinates and the third image coordinates,scanning the image in a fourth direction to determine a fourth set ofpixels, the fourth set of pixels being positioned along a fourth pathbetween the third image coordinates and the second image coordinates;and determining a first intersection point, a second intersection point,a third intersection point, and a fourth intersection point based atleast in part on an intersection of the first set of pixels, the secondset of pixels, the third set of pixels, and the fourth set of pixels.

A scan line can be, for example, a single row of pixels in the image. Inthis example, the image can be divided into a plurality of scan lines(i.e., rows of pixels), and the pixels can analyzed to located theoutermost pixels within the region of interest. For example, based onthe pixels value at each location, a transition from a first pixel value(e.g., 1) to a series of a second pixel value (e.g., 0) can bedetermined. The last pixel value before the transition can be theoutermost pixel of the image at that location. This can be repeated foreach scan line of the image to determine the outermost pixels within theregion of interest. It should be noted that any one of a number ofapproaches can be used to determine the outermost pixels, and anyapproaches described are for purposes of explanation. As shown in FIG.2G, the outermost pixels correspond to the solid adjacent borders, the“L” shape, and the two other borders consisting of the alternating darkand light modules.

The outermost pixels can be divided into four groups. For example, asshown in example 280 of FIG. 2J, the outermost pixels are divided intofour groups by lines AD, DB, BC, and CA. Using at least one line fittingalgorithm, a straight line can be determined for each group of pixels.As would be understood to one skilled in the art, line fitting is theprocess of constructing a line, curve, or mathematical function, thathas the best fit to a series of data points. Line fitting can involveeither interpolation, where an exact fit to the data is required, orsmoothing, in which a “smooth” function is constructed thatapproximately fits the data. As shown in FIG. 2H, the four straightlines interest at four points, A′, B′, C′, and D′. The four points (292,294, 296, and 298) correspond to pixels that are associated with pixelslocations in one of the four groups. In accordance with variousembodiments, the four points can represent corners of the candidate datamatrix 299.

For each candidate data matrix, at least two tests can be performed suchas an aspect ratio test and a timing pattern test. In this example,there is one candidate data matrix. It should be noted that variousother tests can be used to verify whether a data matrix exists, as inknown to those skilled in the art.

The aspect ratio test can be used to verify a ratio of, for example, alength and width of the candidate data matrix. If the ratio of thelength and width of the candidate data matrix meet an aspect ratiothreshold within a certain deviation, the test is successfully passed.

For example, a ratio of the length of the segment formed by A′ and D′ tothe length of segment A′ and C′ can be determined and the ratio can becompared to an aspect ratio threshold (e.g., 1.0). It should be notedthat in this example the geometric shape of the candidate data matrixcorresponds to a square. However, as would be understood to thoseskilled in the art, the geometric shape can corresponds to other shapes,as may include a rectangle, etc. In accordance with various embodiments,there can be one or more aspect ratio thresholds. For example, a firstthreshold can be 1.0 and a second threshold can be 0.50. The determinedratio of the candidate data matrix can be compared to each of theseaspect ratio thresholds. If the ratio of the candidate data matrix meetsat least one of these thresholds within a certain deviation, the aspectratio test is passed.

The timing pattern test, or other like test, can be used to verify aparticular pattern represented in the candidate data matrix. If theparticular pattern is identified and the aspect ratio test is passed, itcan be assumed the region of interest includes a data matrix. Thereafterthe data matrix can be processed, as may include decoding the datamatrix, providing the data matrix to another computing device, etc.

In this example, the borders along lines AD′ and DB′ can be analyzed toidentify a black/white pixel ratio. The pixel ratio for each border(e.g., the pixel ratio for line AD′ and the pixel ratio for line DB′)can be compared to a pixel ratio threshold. As example pixel ratiothreshold can be, for example, 0.5. In this example, if the first ratioassociated with line AD′ passes the pixel ratio test, and the secondratio associated with line DB′ passes the pixel ratio test, the timingpattern test can be considered passed. In various embodiments, eachborder can be analyzed. This can include analyzing borders AD′, DB′,BC′, and CA′. If at least two adjacent borders are associated with apixel ratio that at least meets the pixel ratio test, then the timepattern test is passed. It should be noted that various other pixelratio thresholds can be used as well as other requirements where more orfewer borders must pass the pixel ratio test.

In the situation where both the aspect ratio test and the timing patterntest are passed, it can be assumed the region of interest includes avalid data matrix. Using the location of points A′, B′, C′, and D′ thearea of interest that includes the data matrix can be rectified. Thatis, given a first view of the data matrix from a first viewing positionand viewing angle as seen in FIG. 2A, a second view of the data matrixcan be determined. Such a view is illustrated in example 295 of FIG. 2K,where the region of interest is rectified to illustrate the second viewof the data matrix. In the second view, the data matrix is appropriatelyoriented for a particular decoder component. In accordance with anembodiment, the rectified region of interest can be cropped from theimage and the cropped region of interest that includes the data matrixcan be provided to a decoding component. In another example, coordinateinformation describing the rectified location of the region of intereston the image can be determined and the image and the coordinateinformation can be provided to a decoding component. In certainembodiments, various image processing approaches can be performed beforeproviding information identifying the data matrix or before providingthe data matrix to a decoding component or process.

Accordingly, in accordance with various embodiments, approaches enable aperson in a store who wants to obtain information about an item, such asreviews for a book or a price from a retailer of that item can use anapplications to capture an image of a quick response (QR) code on theitem, and upload that image to a central server (or process the imagelocally on the device) where the image can be analyzed and decodedregardless that data matrix may be positioned at an arbitrary positionand/or orientation in the image.

FIG. 3 illustrates an example process for identifying visual codes froma captured image that are positioned at an arbitrary position and/ororientation in the captured image in accordance with variousembodiments. In this example, an image is obtained 302 that allegedlycontains at least one visual code. As discussed, the image can becaptured or uploaded by a user, or otherwise obtained. As described, insome situations, the data matrix may be positioned such that it isdifficult to align and orient the data matrix in the center or otherappropriate area of the image. Further, a device such as a tabletcomputer or smart phone often is held in a user's hand, and thus mightbe moving slightly while capturing the image, which can make itdifficult to align and orient the data matrix. Further still, any of anumber of other reasons might lead to an arbitrary position and/ororientation of the data matrix in the captured image, which may make itnot possible or at least difficult for conventional data matrix readingalgorithms to provide an accurate result in at least some cases.

In at least some embodiments, one or more preprocessing steps areperformed 304 for the obtained image. For example, the image to beanalyzed can be converted to a binarized or grayscale (or othermonochrome) image, color normalization can be performed, and other suchimage preprocessing can be performed as desired. In accordance with anembodiment, the binarized image can include a plurality of pixels(“image elements”), each pixel of the plurality of pixels beingassociated with one of a first pixel value or a second pixel value. Asdescribed, the pixels values can be, for example 1 and 0.

The image is analyzed 306 to determine at least one connected region ofpixels. A connected region, path of adjacent pixels, connectedcomponent, can be determined by, for example, using aconnected-component detection algorithm. The connected region of pixelscan be analyzed 308 to determine a pair of pixels, included in theconnected region of pixels, that is separated a greatest distance,wherein a first pixel and second pixel of the pair of pixels isassociated with image coordinates. Using the image coordinates of thepair of pixels, a region (or area) of the image that includes apotential (or candidate) data matrix can be determined 310. For example,a third pixel in the connected region of pixels can be identified, wherethe third pixel is associated with a third location that is a furthestlocation from both of the first location and the second location thanany pixel in the connected region of pixels. Identifying the third pixelcan include, for example, determining, for each pixel in the path ofadjacent pixels, a sum of distance to the first location and the secondlocation, and selecting a largest sum of distance, the largest sum ofdistance corresponding to the third pixel. A fourth location of a fourthpixel can be estimated using a geometric relationship between the firstlocation, the second location, and the third location and a region ofinterest can be determined based at least in part on the first location,second location, third location, and fourth location.

The region of interest can be analyzed to verify that the regionincludes the data matrix. The verification stage can include at leasttwo tests, an aspect ratio test 312 and a timing pattern test 314. Theaspect ratio test can be used to verify a ratio of, for example, alength and width of the candidate data matrix. If the ratio of thelength and width of the candidate data matrix meet an aspect ratiothreshold within a certain deviation, the test is successfully passedand the timing pattern test is performed. The timing pattern test, orother like test, can be used to verify a particular pattern representedin the candidate data matrix. If the particular pattern is identifiedand the aspect ratio test is passed, it can be assumed the region ofinterest includes a data matrix. In the situation where the presence ofthe data matrix is verified, the data matrix can be processed 316. Forexample, the region of interest can be cropped from the image and thecropped region of interest that includes the data matrix can be providedto a decoding component. In another example, coordinate informationdescribing the location of the region of interest on the image can bedetermined and the image and the coordinate information can be providedto a decoding component. In certain embodiments, various imageprocessing approaches can be performed before providing informationidentifying the data matrix or before providing the data matrix to adecoding component or process. In at least some embodiments, thedecoding process can be aborted if either of the tests fail. In thissituation the process repeats with a different connected region ofpixels.

Using processes discussed and suggested herein, a computing devicehaving access to a decoding algorithm can capture and analyze an imageof a data matrix, in order to determine information encoded in the datamatrix. This information then can be used for any of a number ofpurposes as discussed elsewhere herein. In one example, a user might seea product or item of interest, such as a book, and might take a pictureof a data matrix on the cover of the book in order to obtain informationabout the book, such as related books, user reviews, or other suchinformation. The user can utilize any appropriate image capture elementon the device, such as a digital still or video camera.

In some cases, a process in accordance with a given embodiment might notprovide acceptable accurate results due at least in part to some aspectof the data matrix and/or captured image. In at least some embodiments,the identifying process can continue with additional steps to attempt tofind a match while accounting for a number of potential aspects orissues with the captured information. For example, the process mightperform additional steps to attempt to handle other barcode formats,such as UPC-E or EAN-8. The process also might attempt to correct forany noise in a central set of rows or other portion of the image, suchas where there were specular reflections in the captured image or thedata matrix was damaged in some way. The process might also attempt toaddress the fact that the data matrix might be placed on a curved orother non-flat surface, such as where the data matrix is on a bottle orcan.

For other data matrix formats, many of the steps or approaches describedabove can be applied as well. In some embodiments, the process can firstbe run for a most common data matrix format, and then can be re-run foreach of a number of other less common formats until a match is found orit is determined that the image cannot be decoded. In other embodiments,certain steps of the process can be performed multiple times fordifferent formats, such that the entire process does not have to berepeated in sequence for each potential format until a match is found.Similarly, when favorable results cannot be obtained using a givenconnected-component, additional connected-component can be analyzed forthe same image. These additional connected-component attempts can bedone serially as well, or multiple connected-components can be analyzedat the same time in order to decrease the average result time, althoughsuch an approach would also increase the average amount of resourcesneeded to find a result. Similar additional processing can be requiredto address the potential for a curved data matrix. Fortunately, theprocesses described herein can be performed quickly, such that multiplevariations of the data matrix decoder can be executed concurrently inorder to attempt to capture any anticipated variation. Variouscombinations of the processes can be utilized for each incoming image toattempt to obtain the most accurate data matrix results.

FIG. 4 illustrates an example of a computing device 400 that can be usedin accordance with various embodiments. Although a portable computingdevice (e.g., a smart phone, an electronic book reader, or tabletcomputer) is shown, it should be understood that any device capable ofreceiving and processing input can be used in accordance with variousembodiments discussed herein. The devices can include, for example,desktop computers, notebook computers, electronic book readers, personaldata assistants, cellular phones, video gaming consoles or controllers,television set top boxes, and portable media players, among others.

In this example, the computing device 400 has a display screen 402,which under normal operation will display information to a user facingthe display screen (e.g., on the same side of the computing device asthe display screen). The computing device in this example can includeone or more image capture elements, in this example including one imagecapture element 404 on the back side of the device, although it shouldbe understood that image capture elements could also, or alternatively,be placed on the sides or corners of the device, and that there can beany appropriate number of capture elements of similar or differenttypes. Each image capture element 404 may be, for example, a camera, acharge-coupled device (CCD), a motion detection sensor, or an infraredsensor, or can utilize any other appropriate image capturing technology.The computing device can also include at least one microphone or otheraudio capture element(s) capable of capturing other types of input data,as known in the art, and can include at least oneorientation-determining element that can be used to detect changes inposition and/or orientation of the device. Various other types of inputcan be utilized as well as known in the art for use with such devices.

FIG. 5 illustrates a set of basic components of a computing device 500such as the device 400 described with respect to FIG. 4. In thisexample, the device includes at least one processor 502 for executinginstructions that can be stored in a memory device or element 504. Aswould be apparent to one of ordinary skill in the art, the device caninclude many types of memory, data storage or computer-readable media,such as a first data storage for program instructions for execution bythe processor 502, the same or separate storage can be used for imagesor data, a removable memory can be available for sharing informationwith other devices, and any number of communication approaches can beavailable for sharing with other devices. The device typically willinclude some type of display element 506, such as a touch screen,electronic ink (e-ink), organic light emitting diode (OLED) or liquidcrystal display (LCD), although devices such as portable media playersmight convey information via other means, such as through audio speakersor another audio element 510. As discussed, the device in manyembodiments will include at least one image capture element 508, such asat least one ambient light camera that is able to image a user, people,or objects in the vicinity of the device. An image capture element caninclude any appropriate technology, such as a CCD image capture elementhaving a sufficient resolution, focal range and viewable area, tocapture an image of the user when the user is operating the device.Methods for capturing images or video using an image capture elementwith a computing device are well known in the art and will not bediscussed herein in detail. It should be understood that image capturecan be performed using a single image, multiple images, periodicimaging, continuous image capturing, image streaming, etc.

The device can include at least one additional input device 512 able toreceive conventional input from a user. This conventional input caninclude, for example, a push button, touch pad, touch screen, wheel,joystick, keyboard, mouse, trackball, keypad or any other such device orelement whereby a user can input a command to the device. These I/Odevices could even be connected by a wireless infrared or Bluetooth orother link as well in some embodiments. In some embodiments, however,such a device might not include any buttons at all and might becontrolled only through a combination of visual and audio commands suchthat a user can control the device without having to be in contact withthe device.

As discussed, different approaches can be implemented in variousenvironments in accordance with the described embodiments. For example,FIG. 6 illustrates an example of an environment 600 for implementingaspects in accordance with various embodiments. As will be appreciated,although a Web-based environment is used for purposes of explanation,different environments may be used, as appropriate, to implement variousembodiments. The system includes an electronic client device 602, whichcan include any appropriate device operable to send and receiverequests, messages or information over an appropriate network 604 andconvey information back to a user of the device. Examples of such clientdevices include personal computers, cell phones, handheld messagingdevices, laptop computers, set-top boxes, personal data assistants,electronic book readers and the like. The network can include anyappropriate network, including an intranet, the Internet, a cellularnetwork, a local area network or any other such network or combinationthereof. Components used for such a system can depend at least in partupon the type of network and/or environment selected. Protocols andcomponents for communicating via such a network are well known and willnot be discussed herein in detail. Communication over the network can beenabled via wired or wireless connections and combinations thereof. Inthis example, the network includes the Internet, as the environmentincludes a Web server 606 for receiving requests and serving content inresponse thereto, although for other networks, an alternative deviceserving a similar purpose could be used, as would be apparent to one ofordinary skill in the art.

The illustrative environment includes at least one application server608 and a data store 610. It should be understood that there can beseveral application servers, layers or other elements, processes orcomponents, which may be chained or otherwise configured, which caninteract to perform tasks such as obtaining data from an appropriatedata store. As used herein, the term “data store” refers to any deviceor combination of devices capable of storing, accessing and retrievingdata, which may include any combination and number of data servers,databases, data storage devices and data storage media, in any standard,distributed or clustered environment. The application server 608 caninclude any appropriate hardware and software for integrating with thedata store 610 as needed to execute aspects of one or more applicationsfor the client device and handling a majority of the data access andbusiness logic for an application. The application server providesaccess control services in cooperation with the data store and is ableto generate content such as text, graphics, audio and/or video to betransferred to the user, which may be served to the user by the Webserver 606 in the form of HTML, XML or another appropriate structuredlanguage in this example. The handling of all requests and responses, aswell as the delivery of content between the client device 602 and theapplication server 608, can be handled by the Web server 606. It shouldbe understood that the Web and application servers are not required andare merely example components, as structured code discussed herein canbe executed on any appropriate device or host machine as discussedelsewhere herein.

The data store 610 can include several separate data tables, databasesor other data storage mechanisms and media for storing data relating toa particular aspect. For example, the data store illustrated includesmechanisms for storing content (e.g., production data) 612 and userinformation 616, which can be used to serve content for the productionside. The data store is also shown to include a mechanism for storinglog or session data 614. It should be understood that there can be manyother aspects that may need to be stored in the data store, such as pageimage information and access rights information, which can be stored inany of the above listed mechanisms as appropriate or in additionalmechanisms in the data store 610. The data store 610 is operable,through logic associated therewith, to receive instructions from theapplication server 608 and obtain, update or otherwise process data inresponse thereto. In one example, a user might submit a search requestfor a certain type of item. In this case, the data store might accessthe user information to verify the identity of the user and can accessthe catalog detail information to obtain information about items of thattype. The information can then be returned to the user, such as in aresults listing on a Web page that the user is able to view via abrowser on the user device 602. Information for a particular item ofinterest can be viewed in a dedicated page or window of the browser.

Each server typically will include an operating system that providesexecutable program instructions for the general administration andoperation of that server and typically will include computer-readablemedium storing instructions that, when executed by a processor of theserver, allow the server to perform its intended functions. Suitableimplementations for the operating system and general functionality ofthe servers are known or commercially available and are readilyimplemented by persons having ordinary skill in the art, particularly inlight of the disclosure herein.

The environment in one embodiment is a distributed computing environmentutilizing several computer systems and components that areinterconnected via communication links, using one or more computernetworks or direct connections. However, it will be appreciated by thoseof ordinary skill in the art that such a system could operate equallywell in a system having fewer or a greater number of components than areillustrated in FIG. 6. Thus, the depiction of the system 600 in FIG. 6should be taken as being illustrative in nature and not limiting to thescope of the disclosure.

The various embodiments can be further implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially-available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially-available protocols, such as TCP/IP, OSI, FTP,UPnP, NFS, CIFS and AppleTalk. The network can be, for example, a localarea network, a wide-area network, a virtual private network, theInternet, an intranet, an extranet, a public switched telephone network,an infrared network, a wireless network and any combination thereof.

In embodiments utilizing a Web server, the Web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more Web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C# or C++ or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch-sensitive displayelement or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random access memory (RAM) orread-only memory (ROM), as well as removable media devices, memorycards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs such as a client applicationor Web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and computer readable media for containing code, orportions of code, can include any appropriate media known or used in theart, including storage media and communication media, such as but notlimited to volatile and non-volatile, removable and non-removable mediaimplemented in any method or technology for storage and/or transmissionof information such as computer readable instructions, data structures,program modules or other data, including RAM, ROM, EEPROM, flash memoryor other memory technology, CD-ROM, digital versatile disk (DVD) orother optical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices or any other medium which canbe used to store the desired information and which can be accessed by asystem device. Based on the disclosure and teachings provided herein, aperson of ordinary skill in the art will appreciate other ways and/ormethods to implement the various embodiments.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

What is claimed is:
 1. A computing device, comprising: a computingdevice processor; a memory device including instructions that, whenexecuted by the computing device processor, cause the computing deviceto: obtain an image including a visual code, the visual code being oneof a plurality of format types and having an arbitrary orientation withrespect to the image; analyze the image to determine a connected regionof pixels of a plurality of pixels; determine a first pixel of theconnected region of pixels associated with first pixel coordinates and asecond pixel associated with second pixel coordinates; determine aregion of the image that includes the visual code based at least in parton the first pixel coordinates and the second pixel coordinates;determine a plurality of borders associated with the region; determine ageometric shape of the region based at least in part on the plurality ofborders of the region, the geometric shape having a length and a width;determine a ratio of the length and the width of the geometric shape;determine whether the ratio meets at least one aspect ratio threshold;verify the region includes the visual code when the ratio of the lengthand the width meets the at least one aspect ratio threshold; determinethe format type of the visual code; and rectify the visual code from thearbitrary orientation to a rectified orientation suitable for a decoderfor the format type; and decode the visual code using the decoder. 2.The computing device of claim 1, wherein the instructions, when executedfurther enable the computing device to: determine a first segment and asecond segment of the region; determine a first pattern of pixels forthe first segment; determine a second pattern of pixels for the secondsegment; and determine that the first pattern of pixels and the secondpattern of pixels satisfies a ratio of first pixels values to secondpixel values.
 3. The computing device of claim 1, wherein theinstructions, when executed to decode the visual code further enable thecomputing device to: crop the potential region from the image togenerate a cropped image that includes the visual code; and provide thevisual code to a decoding component.
 4. The computing device of claim 1,wherein the instructions, when executed to decode the visual codefurther enable the computing device to: determine coordinate informationdescribing the region of the image; and provide the image and thecoordinate information to a decoding component.
 5. The computing deviceof claim 1, wherein the first pixel and the second pixel are part of apair of pixels of the connected region of pixels, and wherein the pairof pixels is separated a greatest distance of the connected region ofpixels.
 6. The computing device of claim 5, wherein the instructionswhen executed further enable the computing device to: for individualpixels in the connected region of pixels, determine a distancemeasurement to pixels in the connected region of pixels to generate aplurality of distance measurements; and select the pair of pixelsassociated with a greatest distance measurement from the plurality ofdistance measurements.
 7. The computing device of claim 1, wherein theinstructions when executed to analyze the image further enable thecomputing device to: convert the image to a binarized image, thebinarized image including a plurality of pixels, individual pixels ofthe plurality of pixels being associated with one of a first pixel valueor a second pixel value; and analyze the binarized image using one of aconnected-component analysis algorithm, a blob extraction algorithm, aregion labeling algorithm, a blob discovery algorithm, or a regionextraction algorithm to identify the connected region of pixels.
 8. Thecomputing device of claim 1, wherein the instructions when executed toconvert the image further enable the computing device to: normalizecolor values for individual color channels of individual pixels to avalue between 0 and
 1. 9. The computing device of claim 1, wherein theinstructions when executed to determine the region of the image furtherenable the computing device to: for individual pixels in the connectedregion of pixels, determine a first distance measurement to the firstpixel coordinates, determine a second distance measure to the secondpixel coordinates, determine a sum of the first distance measure and thesecond distance measure to generate a plurality of distance sums, selecta greatest distance sum of the plurality of distance sums, the greatestdistance sum being associated with a third pixel associated with thirdpixel coordinates in the connected region of pixels; and estimate fourthpixel coordinates of a fourth pixel using a geometric relationshipbetween the first pixel coordinates, the second pixel coordinates, andthe third pixel coordinates.
 10. The computing device of claim 9,wherein the instructions when executed further enable the computingdevice to: scan the plurality of pixels in a first direction todetermine a first set of pixels, the first set of pixels beingpositioned along a first path between the first pixel coordinates andthe second pixel coordinates; scan the plurality of pixels in a seconddirection to determine a second set of pixels, the second set of pixelsbeing positioned along a second path between the first pixel coordinatesand fourth pixel coordinates; scan the plurality of pixels in a thirddirection to determine a third set of pixels, the third set of pixelsbeing positioned along a third path between the fourth pixel coordinatesand the third pixel coordinates, scan the plurality of pixels in afourth direction to determine a fourth set of pixels, the fourth set ofpixels being positioned along a fourth path between the third pixelcoordinates and the second pixel coordinates; and determine a firstintersection point, a second intersection point, a third intersectionpoint, and a fourth intersection point based at least in part on anintersection of the first set of pixels, the second set of pixels, thethird set of pixels, and the fourth set of pixels.
 11. A method,comprising: obtaining an image including a visual code, the visual codebeing one of a plurality of format types and having an arbitraryorientation with respect to the image; analyzing the image to determinea connected region of pixels of a plurality of pixels; determining afirst pixel of the connected region of pixels associated with firstpixel coordinates and a second pixel associated with second pixelcoordinates; determining a region of the image that includes the visualcode based at least in part on the first pixel coordinates and thesecond pixel coordinates; determining a first border and a second borderassociated with the region; determining a ratio of a first length of thefirst border and a second length of the second border; verifying theregion includes the visual code based at least in part on whether theratio meets at least one aspect ratio threshold; determining the formattype of the visual code; and rectifying the visual code from thearbitrary orientation to a rectified orientation suitable for a decoderfor the format type; and decoding the visual code using the decoder. 12.The method of claim 11, further comprising: for individual pixels in theconnected region of pixels, determining a distance measurement to pixelsin the connected region of pixels to generate a plurality of distancemeasurements; and selecting a pair of pixels associated with a greatestdistance measurement from the plurality of distance measurements, thepair of pixels including the first pixel and the second pixel.
 13. Themethod of claim 11, wherein determining the region of the image furtherincludes: for individual pixels in the connected region of pixels,determining a first distance measurement to the first pixel coordinates,determining a second distance measure to the second pixel coordinates,determining a sum of the first distance measure and the second distancemeasure to generate a plurality of distance sums, selecting a greatestdistance sum of the plurality of distance sums, the greatest distancesum being associated with a third pixel associated with third pixelcoordinates in the connected region of pixels; and estimating fourthpixel coordinates of a fourth pixel using a geometric relationshipbetween the first pixel coordinates, the second pixel coordinates, andthe third pixel coordinates.
 14. The method of claim 11, furthercomprising: converting the image to a binarized image, the binarizedimage including a plurality of pixels, each pixel of the plurality ofpixels being associated with one of a first pixel value or a secondpixel value; and analyzing the binarized image using one of aconnected-component analysis algorithm, a blob extraction algorithm, aregion labeling algorithm, a blob discovery algorithm, or a regionextraction algorithm to identify the connected region of pixels.
 15. Themethod of claim 11, further comprising: verifying that the regionincludes the visual code using one of a first verification test and asecond verification test.
 16. A non-transitory computer readable storagemedium storing one or more sequences of instructions executable by oneor more processors to perform a set of operations comprising: obtainingan image including a visual code, the visual code being one of aplurality of format types and having an arbitrary orientation withrespect to the image; analyzing the image to determine a connectedregion of pixels of a plurality of pixels; determining a first pixel ofthe connected region of pixels associated with first pixel coordinatesand a second pixel associated with second pixel coordinates; determininga region of the image that includes the visual code based at least inpart on the first pixel coordinates and the second pixel coordinates;determining a plurality of borders associated with the region;determining a geometric shape of the region based at least in part onthe plurality of borders of the region, the geometric shape having alength and a width; determining a ratio of the length and the width ofthe geometric shape; determining whether the ratio meets at least oneaspect ratio threshold; verifying the region includes the visual codewhen the ratio of the length and the width meets the at least one aspectratio threshold; determining the format type of the visual code;rectifying the visual code from the arbitrary orientation to a rectifiedorientation suitable for a decoder for the format type; and decoding thevisual code using the decoder.
 17. The non-transitory computer readablestorage medium of claim 16, further comprising instructions executed bythe one or more processors to perform the set of operations of: forindividual pixels in the connected region of pixels, determining adistance measurement to pixels in the connected region of pixels togenerate a plurality of distance measurements; and selecting a pair ofpixels associated with a greatest distance measurement from theplurality of distance measurements, the pair of pixels including thefirst pixel and the second pixel.
 18. The non-transitory computerreadable storage medium of claim 16, further comprising instructionsexecuted by the one or more processors to perform the set of operationsof: for individual pixels in the connected region of pixels, determininga first distance measurement to the first pixel coordinates, determininga second distance measure to the second pixel coordinates, determining asum of the first distance measure and the second distance measure togenerate a plurality of distance sums, selecting a greatest distance sumof the plurality of distance sums, the greatest distance sum beingassociated with a third pixel associated with third pixel coordinates inthe connected region of pixels; and estimating fourth pixel coordinatesof a fourth pixel using a geometric relationship between the first pixelcoordinates, the second pixel coordinates, and the third pixelcoordinates.
 19. The non-transitory computer readable storage medium ofclaim 16, further comprising instructions executed by the one or moreprocessors to perform the set of operations of: cropping the region fromthe image to generate a cropped image that includes the visual code; ordetermining coordinate information describing the region of the image;and decoding the visual code.
 20. The non-transitory computer readablestorage medium of claim 16, further comprising instructions executed bythe one or more processors to perform the set of operations of:determining a first segment and a second segment of the region;determining a first pattern of pixels for the first segment; determininga second pattern of pixels for the second segment; and determining thatthe first pattern of pixels and the second pattern of pixels satisfies aratio of first pixels values to second pixel values.